Aggregation of supports along the Lasso path

1 In linear regression with fixed design, we propose two procedures that aggregate a datadriven collection of supports. The collection is a subset of the 2 possible supports and both its cardinality and its elements can depend on the data. The procedures satisfy oracle inequalities with no assumption on the design matrix. Then we use these procedures to aggregate the supports that appear on the regularization path of the Lasso in order to construct an estimator that mimics the best Lasso estimator. If the restricted eigenvalue condition on the design matrix is satisfied, then this estimator achieves optimal prediction bounds. Finally, we discuss the computational cost of these procedures. Let n, p be two positive integers. We consider the mean estimation problem Yi = μi + ξi, i = 1, ..., n, where μ = (μ1, ..., μn) ∈ R is unknown, ξ = (ξ1, ..., ξn) is a subgaussian vector, that is, E[exp(v ξ)] ≤ exp σ |v|2 2 for all v ∈ R , where σ > 0 is the noise level and | · |2 is the Euclidean norm in R. We only observe y = (Y1, ..., Yn) and wish to estimate μ. A design matrix X of size n×p is given and p may be larger than n. We do not require that the model is well-specified, i.e., that there exists β∗ ∈ R such that μ = Xβ∗. Our goal is to find an estimator μ̂ such that the prediction loss ‖μ̂− μ‖2 is small, where ‖ · ‖2 is the empirical loss defined by ‖u‖2 = 1 n |u|2 = 1 n n ∑ i=1 ui , u = (u1, ..., un) ∈ R. In a high-dimensional setting where p > n, the Lasso is known to achieve good prediction performance. For any tuning parameter λ > 0, define the Lasso estimate β̂ l λ as any solution of the convex minimization problem β̂ l λ ∈ argmin β∈Rp 1 2n |y− Xβ| 2 2 + λ |β|1 , where |β|1 = ∑n j=1 |βj | is the `1-norm. The paper studies an estimator μ̂ that is constructed by aggregation or selection of the Lasso estimators that appear on the regularization path of the Lasso. This estimator μ̂ mimics the best Lasso estimator, in the sense that with probability at least 1− 1/n we have ‖μ̂− μ‖2 ≤ C min λ>0 ( ‖Xβ̂ l λ − μ‖2 + σ2(1 + |β̂ l λ|0) n log ( ep 1 + |β̂ l λ|0 )) , 1. Extended abstract. Full version appears as (Bellec, 2016, v2).